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TITLE OF THE INVENTION 
IMAGE PROCESSING METHOD AND APPARATUS 
5 BACKGROUND OF THE INVENTION 

Field of the Invention 

[0001] The present invention relates to image processing 
techniques. More specifically, the present invention 

10 relates to image processing techniques for reading and 

playing back, developing, or processing image information 
input from an optical sensor such as a CCD or CMOS sensor of 
a digital still camera or digital video camera and recorded 
on a medium. 

15 Description of the Related Art 

[0002] Techniques for using digital image information 
captured by a digital still camera or digital video camera 
and output from an image sensor without executing image 
processing relating to the colors of the image such as 

20 adjustment of white balance have been known. (Hereinafter, 

such digital image information that has not substantially 
been processed will be referred to as RAW image data. ) For 
example, in the case of a RAW image obtained by A/D 
conversion of signals output from an imaging device without 

25 compressing it, a playback or development process is always 
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executed based on the RAW image. Thus, for example, 
sophisticated image processing can be executed based on 
parameters arbitrarily set by a user using an image 
processing unit external or internal to the imaging device, 
5 Accordingly, image data including luminance signals and 

color signals is played back or developed, serving to 
achieve a higher quality. Furthermore, depending on the 
preference of the user, image data for playback or 
development that are not affected by degradation of image 
10 quality of image processing can be generated as many times 

as desired. 

[0003] When a RAW image is processed and/or played back 
by an image processing apparatus, image processing such as 
interpolation of pixels, adjustment of white balance, 

15 adjustment of luminance, adjustment of colors, and 

adjustment of sharpness are executed before playback, based 
on property information recorded in association with the RAW 
image or characteristics of main image information. For 
example, the image is expanded in accordance with the 

20 compression format, color interpolation of pixels and 

adjustment of white balance are executed based on the 
property of the main image information and information 
regarding a light source used for imaging, and adjustment of 
luminance, adjustment of colors, and adjustment of sharpness 

25 are executed based on exposure time, shutter speed, zoom 
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position, and other related parameters. 

[0004] Furthermore, when image processing such as color 
adjustment is executed on image information that has been 
played back, image processing and/or playback is executed 
5 based on property information and user-defined information. 

The color filter characteristics refer to characteristics 
including at least one of pixel arrangement , arrangement of 
color filters, sensitivity of pixels (e.g., dispersion 
characteristics) . 

10 [0005] However, property information of RAW images, 

needed for image processing and/or playback, varies 
depending on manufacturers and device types. Furthermore, 
due to difference in color filter characteristics (color 
filter arrangement, distinction between complementary color 

15 filters and primary color filters, distinction between a 

sensor including multi-color photosensitive -layers having 
color separation characteristics with respect to the depth 
direction and a sensor including monochrome photosensitive- 
layers, bit depth, aspect ratio of pixels) of CCDs and 

20 difference in compression formats, different image 

processing methods, and therefore different image processing 
apparatuses, must be used depending on characteristics of 
RAW images specific to different camera types. The bit 
depth refers to the number of bits used to represent the 

25 tone corresponding to an image signal from the CCD or the 
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number of colors. The aspect ratio of pixels is the ratio 
of the vertical side to the horizontal side of a rectangular 
array of pixels. 

[0006] Thus, in order to allow operations on such RAW 
5 image files by a common image processing apparatus, image 

data are converted into a common format, such as bitmap 
format or JPEG format and then saved. 

[0007] However, when image data that has been converted 
into a common image data format is used, the following 
10 problems arise. 

(1) Even when the dynamic range of an optical sensor is 
wider and the bit depth is deeper accordingly, image data 
must be converted into a common image format in order to 
allow image processing and/or playback by a common image 

15 processing apparatus. This could cause degradation in image 

quality due to limitation in the bit depth, attributable to 
conversion into the common image format. 

(2) When lossy compression is applied during the conversion 
into the common image data format, the image data is 

20 degraded even further. 

(3) Information representing the status of the imaging 
device used for image processing (e.g., ISO sensitivity and 
exposure time) could be lost or erased during conversion 
into the common image data format. 

25 (4) Even if the problem (3) is overcome, that is, even if 
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the information representing the status of the imaging 
device used for image processing (e.g., ISO sensitivity and 
exposure time) is recorded in association with main image 
information during conversion into the common image data 
5 format, it is often the case that the information is not 

used or parsed for image processing by the image processing 
apparatus . 

[0008] On the other hand, when a plurality of types of 
image processing apparatuses are used to use RAW images, the 

10 following problems arise. 

(1) The formats of RAW images vary depending on 
manufacturers or types of imaging devices. Thus, it is 
laborious for the user to use different image processing 
apparatuses depending on RAW images . 

15 (2) When an algorithm more favorable than a currently 

available algorithm has been developed, if different image 
processing apparatuses are used depending on device types, 
developers must alter a substantial amount of code, causing 
an increase in development cost . 

20 (3) When an algorithm more favorable than a currently 

available algorithm has been developed, a user is not 
allowed to adjust image quality of or playback a RAW image 
captured by an imaging device of a manufacturer that does 
not employ the favorable algorithm. 



25 
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SUMMARY OF THE INVENTION 



[0009] The present invention has been made in view of the 
problems described above, and it is an object of the present 
5 invention to provide image data having unified data formats 

of property information and main image information while 
maintaining advantages of RAW images, allowing unification 
of image processing for converting, for example, RAW image 
files into files in common formats. 

10 [0010] In order to achieve the object described above, 

the present invention, in one aspect thereof, provides an 
image processing method for generating image data in a 
second format from image data in a first format, the image 
data in the first format including image data that is 

15 uncompressed or compressed and substantially losslessly, 

obtained by digitizing signals output from an imaging device, 
and the image data in the second format including a 
luminance signal and color signals, the image processing 
method including selecting a processing module to be used 

20 from among a plurality of types of processing modules, based 

on image property information associated with the image data 
in the first format; converting the image data in the first 
format into image data in a third format that is different 
from the second format, using the processing module selected 

25 in the selecting; and generating the image data in the 
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second format based on the Image data in the third format, 
[0011] The present invention, in another aspect thereof, 
provides an image processing apparatus for generating image 
data in a second format from image data in a first format, 
5 the image data in the first format including image data that 

is uncompressed or compressed and substantially losslessly, 
obtained by digitizing signals output from an imaging device, 
and the image data in the second format including a 
luminance signal and color signals, the image processing 

10 apparatus including a selecting unit which selects a 

processing module to be used from among a plurality of types 
of processing modules, based on image property information 
associated with the image data in the first format; a 
converting unit which converts the image data in the first 

15 format into image data in a third format that is different 

from the second format, using the processing module selected 
by the selecting unit; and a generating unit which generates 
the image data in the second format based on the image data 
in the third format. 

20 [0012] The present invention, in another aspect thereof, 

provides a storage medium storing a control program for 
allowing a computer to execute image processing for 
generating image data in a second format from image data in 
a first format, the image data in the first format including 

25 image data that is uncompressed or compressed and 
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substantially losslessly, obtained by digitizing signals 
output from an imaging device, and the image data in the 
second format including a luminance signal and color signals, 
wherein a processing module to be used is selected from 
5 among a plurality of types of processing modules, based on 

image property information associated with the image data in 
the first format, the image data in the first format is 
converted into image data in a third format that is 
different from the second format, using the processing 

10 module selected, and the image data in the second format is 

generated based on the image data in the third format . 
[0013] The present invention, in another aspect thereof, 
provides a control program for allowing a computer to 
execute image processing for generating image data in a 

15 second format from image data in a first format, the image 

data in the first format including image data that is 
uncompressed or compressed and substantially losslessly, 
obtained by digitizing signals output from an imaging device, 
and the image data in the second format including a 

20 luminance signal and color signals, wherein a processing 

module to be used is selected from among a plurality of 
types of processing modules, based on image property 
information associated with the image data in the first 
format, the image data in the first format is converted into 

25 image data in a third format that is different from the 
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second format, using the processing module selected, and the 
image data in the second format is generated based on the 
image data in the third format. 

[0014] Further objects, features and advantages of the 
5 present invention will become apparent from the following 

description of the preferred embodiments (with reference to 
the attached drawings ) . 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 

[0015] The accompanying drawings, which are incorporated 
in and constitute a part of the specification, illustrate 
embodiments of the invention and, together with the 
description, serve to explain the principles of the 
15 invention. 

[0016] Fig. 1 is a block diagram showing the 
configuration of a digital still camera that serves as an 
image input device. 

[0017] Fig. 2 is a flowchart showing a process for 
20 recording an image by an image input device. 

[0018] Fig. 3 is a schematic diagram of an image file. 
[0019] Figs. 4A and 4B are diagrams showing a pixel 
arrangement of image signals . 

[0020] Fig. 5 is a schematic diagram showing how color 
25 signals are input by a photoelectric converter including 



- 10 - 



multi-color photosensitive layers. 

[0021] Figs. 6A and 6B are schematic diagrams showing 
image signals input from a photoelectric converter whose 
aspect ratio of pixels is not 1:1. 
5 [0022] Fig. 7A is a flowchart showing a process of 

displaying and outputting a RAW image. 

[0023] Fig. 7B is a flowchart showing a process of 
displaying and outputting a RAW image. 
[0024] Fig. 8 is a block diagram showing the 
10 configuration of an information processing apparatus for 

executing image processing according to an embodiment of the 
present invention. 

[0025] Fig. 9 is a block diagram showing the functional 
configuration of image processing according to the 
1 5 embodiment . 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0026] Now, a preferred embodiment of the present 
20 invention will be described with reference to the 

accompanying drawings. The embodiment will be described, as 
an example, in the context of an electronic camera such as a 
digital still camera. 

[0027] Fig. 1 is a block diagram showing the 
25 configuration of a digital still camera that serves as an 
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image input device in this embodiment. Referring to Fig. 1, 
an imaging optical system is formed by an imaging lens 101, 
a shutter diaphragm 102, and other related parts. A 
photoelectric converter (hereinafter referred to as a CCD) 
5 103 converts optical signals of a target, focused by the 

imaging optical system (101 and 102), into electric signals. 
[0028] An analog-digital converter 104 converts an output 
of the CCD 103 (an analog signal) into a digital signal. A 
digital processor 105 processes signals digitized by the 

10 analog-digital converter 104. A CPU 106 executes various 

camera- control operations and signal processing. An 
internal memory 107 includes a read-only memory and a random 
access memory. The read-only memory stores control programs. 
The random access memory is used as a temporary storage 

15 memory during processing by the CPU 106. A compressing 

circuit 108 compresses digital signals to be recorded. An 
interface 109 allows connection with a removable storage 
medium. An interface 110 allows connection with a personal 
computer or the like. The analog-digital converter 104 to 

20 the interface 110 and other related parts constitute a 

signal processing circuit of the digital still camera 
according to this embodiment. A storage medium 111 is a 
removable storage medium that is removable from the digital 
still camera. 

25 [0029] Fig. 2 is a flowchart showing image data 
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processing by the digital still camera according to this 
embodiment. This process is executed by the CPU 106 
executing a control program stored in the internal memory 
107. 

[0030] In step S201, image data is input to the memory. 
In this embodiment, optical signals are converted into 
electric signals by the CCD 103, the electric signals are 
digitized by the analog-digital converter 104, and the 
resulting digital signal serves as the image data. In step 
S202, it is determined whether the image data is to be 
processed before recorded in a common image data format 
(e.g., JPEG). More specifically, in this embodiment, it is 
determined whether a person who captured the image selected, 
at the time of imaging, a mode in which data output from the 
CCD 103 is recorded without being processed (RAW image 
output mode). If the RAW image output mode was selected, 
the procedure proceeds to step S204, and otherwise the 
procedure proceeds to step S203. 

[0031] In step S203, image processing is executed. In 
this embodiment, filtering, color adjustment based on 
information such as white balance and auto focus, and the 
like, are executed. Then, in step S204, it is determined 
whether the image data is to be compressed before being 
recorded. More specifically, in this embodiment, it is 
determined whether the person who captured the image 
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selected a compressed recording mode at the time of imaging. 
The procedure proceeds to step S205 if the compressed 
recording mode was selected, and the image is compressed by 
the compressing circuit 108. The compression method is 
5 either lossless or lossy. In step S206, property 

information is attached to the image data. In this 
embodiment, the property information include information 
such as imaging date/ time, type of camera, whether flash is 
on or off, recording format, and other imaging conditions. 

10 In step S207, the image data is recorded on the removable 

recording medium 111. In this embodiment, the removable 
storage medium 111 is, for example, a CompactFlash® card. 
[0032] In this embodiment, image processing is not 
executed in step S202, and image information recorded 

15 without being compressed in step S206 or image information 

lossless -compressed and recorded in step S205, i.e., RAW 
image, is processed. 

[0033] Fig. 3 is schematic diagram of an image file that 
is created by an image recording method in this embodiment . 

20 Referring to Fig. 3, a property- information area 301 is an 

area where information representing imaging parameters and 
other related information needed for reading, playing back, 
or selecting an image, such as image data size, imaging 
date/ time, type of camera, compression method, and whether 

25 flash is on or off, are stored. Furthermore, as information 
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to be used for photoelectric conversion of an optical image, 
for example, information regarding the optical system for 
forming the optical image, such as exposure time, aperture 
ratio, color temperature of a light source irradiating the 
5 target, characteristics of the photoelectric converter 

element, ISO sensitivity, and focal distance of the lens, 
may be included. A reduced- image-data area 302 is an area 
where reduced image data that is shown when a list is 
displayed is stored. A main -image -data area 303 is an area 
10 where all the image data to be stored in the image file is 

stored. When the RAW image output mode is selected, RAW 
image data or lossless -compressed RAW image data is stored 
in the main -image -data area 303. 

[0034] The data structure of the RAW image described 
15 above varies depending on the hardware configuration and 

other device-specific factors. According to the image 
processing apparatus of this embodiment, when an image data 
file including such RAW image data is converted into an 
image data file in a common format, the image data including 
20 RAW image data is first converted into image data in an 

intermediate format. The image data in the intermediate 
format has a unified data format that does not depend on 
machine type, while maintaining the advantages of RAW image. 
Now, an example of variation of RAW images and conversion 
25 into a unified format will be described. 
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[0035] 

1. Main part of image information 

1-1. Absorption of variation in characteristics of imaging 
devices 

Figs . 4A and 4B are diagram showing an example of pixel 
array of image signals . Data that has been digitized by the 
analog-digital converter 104 and then saved, in a case where 
primary-color filters and monochrome photosensitive layers 
are used in the camera, is divided into blocks as shown in 
Fig. 4A. Each of the blocks is composed of four pixels 
including color signals of red, green, and blue (RGB) , as 
shown in Fig. 4B. In the case of complementary filters, 
each of the blocks is composed of color signals for cyan, 
magenta, and yellow (CMY) . In that case, the arrangement of 
the pixels varies depending on the characteristics of CCD 
color filters. Furthermore, the pixels do not necessarily 
have an aspect ratio of 1:1 (e.g., 2:1), as shown in Fig. 6A. 
Furthermore, the bit depth of pixels constituting an image 
could vary depending on the characteristics of CCD color 
filters . 

[0036] As described above, the recording conditions of 
RAW image stored in the main -image -data area 303 could vary 
depending on device type. An example of such a case and an 
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example of conversion will be described below. 
[0037] 

1-2. Unification of bit depth 
5 If the bit depth of pixels constituting an image 

differs from a bit depth for executing common image 
processing, which can be arbitrarily chosen, the former bit 
depth must be matched with the latter bit depth. For 
example, when the bit depth of pixels constituting an image 
10 is 8 bits and the bit depth for executing common image 

processing is 12 bits, the conversion is done by left- 
shifting all the pixel values by four bits. 
[0038] 

ColR_12bit = (ColR_8bit) << (12-8) 
15 ColB_12bit = (ColB_8bit) << (12-8) 

ColG_12bit = (ColG_8bit) << (12-8) (1) 
[0039] Generally, when a particular bit depth x (bits) is 
converted into a bit depth a (bits) for executing common 
image processing, a being larger than x, conversion is 
20 possible by left shifting by (a-x) (bits). 

[0040] 

1-3. Conversion of data in accordance with characteristics 
of color filters 

25 In this embodiment, the format of RAW images composed 
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of RGB values of respective pixels is unified. Thus, in a 
case where CCD color filters are primary color filters and 
monochrome photosensitive layers are used (e.g., the case 
shown in Figs. 4A and 4B) , an interpolation process that 
5 will be described later is performed on RAW image data 

obtained, whereby image data in an intermediate format for 
executing common image processing is obtained. 
[0041] On the other hand, if the CCD color filters are 
complementary color filters, image information that is input 

10 is interpolated to calculate CMY values of respective pixels . 

Cyan, magenta, and yellow (CMY) color signals for the pixels 
constituting the image are converted into red, green, and 
blue (RGB) color signals for executing common image 
processing, i.e., image data in an intermediate format. In 

15 this case, the conversion is done using the following 

equations . 
[0042] 

R = (Y+M-O/2 
G = (C+M-Y)/2 

20 B = (C+Y-M)/2 (2) 

[0043] Furthermore, as shown in Fig. 5, in the case of a 
RAW image obtained by a CCD including multi-color 
photosensitive layers, a conversion process described below 
is executed. Fig. 5 is a schematic diagram showing how 

25 color signals are input through multi-color photosensitive 
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layers. In the case of the multi-color photosensitive 
layers shown in Fig. 5, a single pixel does not correspond 
to a single color signal, but a first layer represents a 
value (R+G+B) obtained by adding up red, green, and blue 
5 color signals, a second layer represents a value (R+G) 

obtained by adding up red and green color signals, and a 
third layer represents a value (R) for red color signal. 
Let the color signal of the first layer be denoted as coll, 
the color signal of the second layer as col2, and the color 

10 signal of the third layer as col3. Then, the following 

equations are used to convert the color signals coll, col2, 
and col3 into red, green, and blue (RGB) color signals for 
executing common image processing, i.e., image data in an 
intermediate format. 

15 [0044] 

R = col3 

G = col2 - col3 

B = coll - col2 (3) 
[0045] For example, in the case of an imaging device 

20 having a color filter arrangement similar to what is called 

a Beyar arrangement, shown in Figs. 4A and 4B, in a RAW 
image obtained by a primary-color CCD, information of all 
colors does not exist in all the pixels . In order to 
generate (interpolate) red, blue, and green color signals 

25 (colR(x,y), colG(x,y), colB(x,y)) that do not originally 
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exist, for example, the following equations are used. 
[0046] 

colR(x,y) = (colR(x,y-l) + (colR(x,y+l ) ) /2 
colG(x,y) = (colG(x,y-l) + (colG(x,y+l ) ) /2 
5 colB(x,y) = (colB(x,y-l) + (colB(x,y+l ) ) /2 (4) 

[0047] Furthermore, if the intervals of pixels in the 
horizontal direction and the vertical direction (aspect 
ratio) are not the same (i.e., the pixels do not form a 
square array (refer to Figs. 6A and 6B)) because of the 
10 arrangement of a reading circuit of the imaging device, an 

interpolation process is performed so that the array of the 
CCD pixels forms a square array. 

[0048] For example, if the ratio of the pixel interval in 
the horizontal direction (the interval between center points 

15 of sampling) to the ratio of the pixel interval in the 

vertical direction is 1:2 (i.e., the pixel interval in the 
horizontal direction is half the pixel interval in the 
vertical direction), for example, considering in terms of 
luminance signals, output signals of the imaging device are 

20 converted so as to represent a square array, for example, by 

the following calculation. In the example described below, 
pixels of a RAW image of three rows x three columns are 
converted into pixels of six rows x twelve columns to 
generate image data representing a square array. 

25 [0049] For example, with regard to the horizontal 
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direction, the following equations are used to generate data 
of six pixels out(x-2,y), out(x-l,y), out(x,y), out(x+l.y), 
out(x+2,y), out(x+3,y) from data of three pixels in(x-l,y), 
in(x.y), in(x+l,y). 
[0050] 

out(x-2,y) = (4*in(x-l,y) )/4 

out(x-l.y) = (3*in(x-l,y) + l*in(x,y))/4 

out(x.y) = (l*in(x-l,y) + 3*in(x.y))/4 

out(x+l # y) = (l*in(x+l,y) + 3*in(x,y))/4 

out(x+2,y) = (3*in(x+l,y) + l*in(x,y))/4 

out(x+3.y) = (4*in(x+l,y) )/4 (5) 
[0051] With regard to the vertical direction, for example, 
the following equations are used to generate data of 12 
pixels out(x,y-5), out(x,y-4), out(x,y-3), out(x,y-2), 
out(x,y-l), out(x,y), out(x,y+l), out(x,y+2), out(x,y+3), 
out(x,y+4), out(x,y+5), out(x,y+6) from data of three pixels 
in(x,y-l). in(x,y), in(x,y+l). 
[0052] 

out(x,y-5) = (16*in(x,y-l) )/16 
out(x,y-4) = (16*in(x,y-l) )/16 
out(x,y-3) = (15*in(x,y-l) + l*in(x,y ) ) /16 
out(x,y-2) = (ll*in(x,y-l) + 5*in(x.y ) ) /16 
out(x.y-l) = (5*in(x,y-l) + ll*in(x,y) ) /16 
out(x.y) = (l*in(x.y-l) + 15*in(x,y) ) /16 
out(x.y+l) = (l*in(x,y+l) + 15*in(x,y) ) /16 
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out(x,y+2) = (5*in(x,y+l) + ll*in(x / y ) ) /16 
out(x,y+3) = (ll*in(x,y+l) + 5*in(x,y) ) /16 
out(x,y+4) = (15*in(x,y+l) + l*in(x,y) ) /16 
out(x,y+5) = (16*in(x,y+l) )/16 

out(x,y+6) = (16*in(x,y+l) )/16 (6) 
[0053] As described above, the pixels of the RAW images 
having three rows x three columns, shown in Figs. 6A and 6B, 
are converted into pixels of six rows x twelve columns, 
whereby image data representing a square array is generated. 
That is, the aspect ratio is corrected. 

[0054] 

2. Unification of the format of property information 
Of property information that can be used in image 
processing of RAW images, examples of information that could 
differ depending on notation (units or the like) and 
examples of conversion of such information will be described 
below. In the examples described below, the formats of 
property information are unified by converting various 
property values into APEX units . 
[0055] 

Shutter speed: Second APEX 

Shutter speed is converted from a second to an APEX 
unit of shutter speed so that common image processing is 
allowed, using the following equation. 
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ShutterSpeedValue : Tv = -log2 ( Second) (7) 
Aperture value: F-Number -> APEX 

Aperture ratio is converted from a F -Number to an APEX 
unit of aperture value so that common image processing is 
5 allowed, using the following equation. 

ApertureValuerAv = 21og2 (F -Number ) (8) 
Brightness value: B/NK -* APEX 

Brightness value is converted from a B/NK to an APEX 
unit of brightness value so that common image processing is 
10 allowed, using the following equation. 

BrightnessValue:Bv = log2(B/NK) (9) 
[0056] An image processing apparatus according to this 
embodiment for allowing the processing described above will 
be described. Fig. 8 shows an information processing 
15 apparatus according to this embodiment, which allows image 

processing described below by executing a predetermined 
control program, functioning as an image processing 
apparatus . 

[0057] Referring to Fig. 8, a CPU 801 is responsible for 
20 the overall control of the apparatus. A ROM 802 stores 

procedures of operation of the CPU 801 (e.g., a program for 
booting the computer, a program for basic input /output , and 
the like). A RAM 803 functions as a main memory for the CPU 
802. Various programs, including a control program for 
25 allowing image processing described later, are loaded from a 
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hard disk drive 804 or the like to the RAM 803, and are 
executed by the CPU 801. The RAM 803 functions as a work 
area when the CPU 801 executes various processes. 
[0058] A hard disk drive (HDD) 804 and a flexible disk 
5 drive (FDD) 805 are used to save and read application 

programs, data, and libraries. Alternatively or in addition 
to the FDD, an optical or magneto -optical disk drive for CD- 
ROM, MO, DVD, or the like, or a magnetic tape drive such as 
a tape streamer or a DDS may be provided. 

10 [0059] An input device 806 includes, for example, a 

keyboard and a pointing device. A display 807 displays 
various information under the control of the CPU 801. 
[0060] An interface 808 allows image data to be received 
from the interface 110 of the digital still camera shown in 

15 Fig. 1. The interfaces 808 and 110 are typically USB 

interfaces. A storage medium mounting unit (media drive) 
809 allows the storage medium 111 detached from the digital 
still camera shown in Fig. 1 to be mounted thereon so that 
data stored in the storage medium 111 can be read. System 

20 bus 810 (including an address bus, a data bus, and a control 

bus) connect the units described above with each other. 
[0061] Image processing in this embodiment by the 
information processing apparatus configured as described 
above will be described in detail with reference to Figs. 7A, 

25 7B, and 9. 
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[0062] Figs. 7A and 7B are flowcharts showing processes 
in which a RAW image is read from a recording apparatus and 
is displayed on an output apparatus (or data is output to a 
storage medium or the like, or a visual image is output onto 
5 a recording medium) in this embodiment. Fig. 9 is a diagram 

showing functional blocks for allowing conversion into image 
data in an intermediate format in this embodiment. 
[0063] In step S701, a RAW image file is read from a 
recording medium. The recording medium is, for example, the 
10 hard disk 804 of the personal computer, or the storage 

medium 111. Alternatively, a RAW image may be read via a 
network that is not shown. 

[0064] In step S702, property information recorded in the 
property -information area 301 of the RAW image file is read. 

15 The property information includes, for example, bit depth, 

color filter characteristics of the CCD color filters, 
compression format, exposure time, shutter speed, aperture 
value, brightness value, exposure compensation factor, 
minimum F value of the lens, target distance, photometry, 

20 light source, flash on/off, focal length of the lens, F 

number, exposure program, ISP speed rate, and flash 
intensity. 

[0065] In step S703, a property- information converting 
unit 902 converts property information expressed in 
25 different notations (units or the like) into common formats 
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(APEX units in this embodiment) according to the equations 
described earlier. (Hereinafter, property information 
having been converted into the common formats will be 
referred to as general property information.) For example, 
5 shutter speed is converted according to equations (7), 

aperture value is converted according to equations (8), and 
brightness value is converted according to equations (9). 
[0066] In step S704, it is determined whether the RAW 
image data in the main -image -data area 303 is compressed. 

10 If the RAW image data is compressed, the RAW image data is 

expanded in step S705 based on the compression format 
included in the general property information. More 
specifically, a selector 903 forwards the RAW image to an 
expanding unit 904 in accordance with the compression format, 

15 where the RAW image is expanded. Although only one 

expanding unit is shown in Fig. 9, a plurality of types of 
expanding units is provided in accordance with a plurality 
of types of compression formats, and one of the expanding 
units is selected by the selector 903. If the RAW image is 

20 not compressed, the RAW image is provided as it is from the 

selector 903 to a bit-depth converting unit 905. 
[0067] In step S706, the bit-depth converting unit 905 
adjusts the bit depth of the RAW image based on the bit- 
depth information included in the general property 

25 information. The bit depth is adjusted according to 
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equations ( 1 ) . 

[0068] In step S707, it is determined whether the imaging 
device has color separation characteristics with respect to 
the depth direction based on the color filter 
5 characteristics included in the general property information. 

If the imaging device does not have color separation 
characteristics with respect to the depth direction (i.e., 
in the case of monochrome photosensitive layers), color 
interpolation is performed in step S708. Since the RAW 

10 image is composed of the blocks shown in Figs. 4A and 4B, a 

color signal representing only one color is assigned to each 
pixel. Thus, color interpolation is performed. For example, 
for the purpose of color interpolation, linear interpolation 
according to equations (4) described earlier is used. This 

15 is a process executed in a case where an interpolating unit 

907 is selected by a selector 906 shown in Fig. 9. 
[0069] On the other hand, if the color filters have color 
separation characteristics with respect to the depth 
direction in Fig. 5, color signals coll, col2, and col3 are 

20 converted into red, green, and blue (RGB) color signals in 

step S709. An example of conversion from the color signals 
coll, col2, and col3 into the red, green, and blue (RGB) 
color signals is expressed in equations (3). This process 
is executed when the selector 906 selects a multi-color 

25 photosensitive -layer data converting unit 909. 
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[0070] Furthermore, in the case of monochrome 
photosensitive layers, after color interpolation is finished 
(steps S707 and S708), it is determined in step S710 whether 
the color filters are complementary filters based on the 
5 color filter characteristics included in the general 

property information. If it is determined that the color 
filters are complementary filters, in step S711, cyan, 
magenta, and yellow (CMY) color signals are converted into 
red, green, and blue (RGB) color signals. An example of 

10 conversion from cyan, magenta, and yellow (CMY) color 

signals into red, green, and blue (RGB) color signals is 
expressed in equations (2). This corresponds to a process 
executed when the selector 906 selects an interpolating unit 
907' and a YMC converting unit 908. The interpolating unit 

15 907 ' executes the same interpolating process (expressed in 

equations (4)) as the interpolating unit 907. 
[0071] Furthermore, in step S712, it is determined 
whether the aspect ratio of the pixels of the CCD is 1:1. 
If it is determined that the aspect ratio of the pixels of 

20 the CCD is not 1:1, conversion is executed in step S713 so 

that the aspect ratio will be 1:1. An example of conversion 
for achieving an aspect ratio of 1:1 is expressed in 
equations (5) and (6). This process is executed by a 
selector 910 and an aspect-ratio converting unit 911 shown 

25 in Fig. 9. 
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[0072] As described above, RAW image data obtained by 
digitizing signals output from the imaging device, either 
uncompressed or lossless -compressed, is converted into image 
data in an intermediate format that is different from a 
common format, using a processing module selected from a 
plurality of types of processing modules based on image 
property information associated with the RAW image data. By 
this process, image information that allows common image 
processing (image data 912 in the intermediate format) while 
maintaining the advantages of RAW image data is obtained. 
Thus, processing in steps S714 to S721 described below does 
not depend on manufacturers or types of imaging devices . 
[0073] In step S714, information that has been set by the 
user for adjusting image quality (hereinafter referred to as 
user-defined information) is read. The user-defined 
information includes, for example, sharpness, contrast, 
color matrix, tone curve, brightness, color saturation, hue, 
and white balance. 

[0074] In step S715, white balance is adjusted based on 
user-defined information and/or general property information. 
General property information used for adjusting white 
balance includes, for example, information regarding light- 
source, whether flash is on or off, and flash intensity. 
[0075] In step S716, red, green, and blue (RGB) color 
signals are converted into brightness (Y), reddishness (Cr), 
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and bluishness (Cb) (YCrCb) signals. Then, in step S717, 
brightness is adjusted based on user-defined information 
and/or general property information. General property 
information used for adjusting brightness includes, for 
5 example, exposure time, shutter speed, aperture value, 

brightness value, exposure compensation factor, target 
distance, photometry, F number, and exposure program. 
Furthermore, in step S718, color is adjusted based on user- 
defined information and/or general property information. 

10 [0076] In step S719, sharpness is adjusted based on user- 

defined information and/or general property information. 
General property information used for adjusting sharpness 
includes, for example, exposure time, shutter speed, 
aperture value, brightness value, exposure compensation 

15 factor, target distance, photometry, F number, exposure 

program, lens distortion, focal distance of the lens, 
minimum F value of the lens, and ISO speed rate. 
[0077] In step S720, brightness, reddishness, and 
bluishness (YCrCb) signals are converted into red, green, 

20 and blue (RGB) color signals. 

[0078] In step S721, the signals are converted into a 
common format and then saved. The common format is, for 
example, bitmap or JPEG format. 

[0079] Since variation due to manufacturers and device 
25 types can be absorbed by the processing in steps S703 to 
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S713 in this embodiment, image processing in steps S714 to 
S721 can be applied to any RAW image. Thus, when a 
favorable algorithm has been developed, it suffices to 
simply change the processing in steps S714 to S721 or add 
5 processing. 

[0080] Furthermore, when CCD color filters having novel 
characteristics have become available, for the purpose of 
conversion into red, green, and blue (RGB) color signals for 
allowing common image processing in this embodiment, it 

10 suffices to add processing or modify the processing in steps 

S706 to S713. Furthermore, when new characteristic 
information that cannot be processed is included, or in the 
case of a RAW image file that cannot be read, information 
that cannot be processed is additionally written in a header 

15 of the RAW image file so that the information can be dealt 

with later by added or modified processing. That is, a RAW 
image file may be partially processed, handing the remaining 
part later by additional processing, or an unprocessed RAW 
image file that cannot be read may be handled by modified 

20 processing. Accordingly, maximum image processing is 

allowed, allowing image files in new formats to be processed 
using updates such as added or modified processing. 
[0081] Although the property information in this 
embodiment has been described as recorded in a header 

25 attached to an image file, the property information may be 



recorded in a separate file or saved in a different media as 
long as the property information is saved in association 
with image information and can be read by an image 
processing apparatus* 

[0082] As described hereinabove, according to the 
embodiment, in an image processing apparatus for reading and 
playing back image information that has been input from an 
image sensor and recorded without undergoing image 
processing, even when property information needed for image 
processing and/or playback of RAW image data and/or 
characteristics of main part of image information differs, 
common image processing can be used for image processing 
and/or playback. Thus, the user is freed of the concern for 
the difference relating to image processing, so that the 
ease of operation is improved. Furthermore, according to 
the embodiment , an original RAW image is converted into 
image data in a common format without reliance upon hardware 
characteristics in a substantially lossless manner by 
formalizing this way, that is, without losing the 
information of the original RAW image, image quality is 
substantially maintained. Furthermore, in developing image 
processing software, with regard to difference in RAW images, 
it suffices to modify source code for image processing only 
at particular parts, such as code relating to interfaces for 
inputting RAW images in different formats. This serves to 
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reduce development cost. 
Other Embodiments 

[0083] It is to be understood that the objects of the 
present invention can be achieved by providing a system or 
apparatus with a storage medium having recorded thereon 
program code of software for achieving the functions of the 
embodiment described above, allowing a computer of the 
system or apparatus to read and execute the program code 
stored in the storage medium. 

[0084] In this case, the program code read from the 
storage medium itself achieves the functions of the 
embodiment described above, so that the storage medium 
storing the program code constitutes the present invention. 
[0085] The storage medium for storing the program code 
may be, for example, a flexible disk, a hard disk, an 
optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a 
magnetic tape, a non-volatile memory card, or a ROM. 
[0086] Also, it is to be understood that, as well as the 
case where the functions of the embodiment described above 
is achieved by the computer reading and executing the 
program code, it also falls with in the scope of the present 
invention when an OS (operating system) or the like running 
on the computer executes part of or the entire processing 
based on instructions of the program code, whereby the 
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functions of the embodiment described above is achieved. 
[0087] Furthermore, the program code read from the 
storage medium may be written in a memory of a functional 
extension board inserted in a computer or a functional 
5 extension unit connected to a computer so that a CPU or the 

like of the functional extension board or functional 
extension unit executes actual processing in part or in 
entirety based on instructions of the program code, whereby 
the functions of the embodiment described above are achieved. 
10 It is to be understood that this also falls within the 

scopes of the present invention. 

[0088] Although the embodiment has been described in the 
context of an example where the imaging device is a CCD, 
other types of imaging device such as a CMOS imaging device 

15 may be used. 

[0089] Although image data obtained by A/D conversion of 
signals output from an imaging device, without compressing 
it, is used as RAW image data in the embodiment, image data 
that has been lossless -compressed after A/D conversion may 

20 be used. Alternatively, RAW data may be analog signals 

output from an imaging device. Other types of RAW data may 
also be used, for example, A/D-converted image signals 
before adjustment of white balance, A/D-converted image 
signals before color separation into luminance signals and 

25 color signals, or signals output through color filters such 



- 34 - 



as Beyar color filters before executing color interpolation. 
[0090] While the present invention has been described 
with reference to what are presently considered to be the 
preferred embodiments, it is to be understood that the 
5 invention is not limited to the disclosed embodiments. On 

the contrary, the invention is intended to cover various 
modifications and equivalent arrangements included within 
the spirit and scope of the appended claims. The scope of 
the following claims is to be accorded the broadest 
10 interpretation so as to encompass all such modifications and 

equivalent structures and functions. 



